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The Educational Research and Development Unil (ERCU) 
of the Graduate School of Industrial Administration at 
Carnegie-Mellon University was formally established in January 1970 
by a S'-year grant from the Ford Foundation* This report describes the 
research and development activities in which ERDU was engaged from 
January to December 1972 for the Ford Foundation* The principal 
activity daring that year was the further development and programiring 
for the Educational Assembly System* (EAS) , a system that can 
generate information about educational materials organized in a 
curriculum best suited to each individual user* The data base of the 
system consists of 2 parts: (a) structured descriptions of 
educational goals and materials, and (b) a network of work 
relationships. Both of these are created by various subject matter 
experts* The user inputs his goal, including information about the 
area, level, time he wants to spend, etc., and the system interacts 
with questions abott possible inconsistencies^ prerequisites and 
other relevant information* Finally the student is presented with the 
optimal curriculum* (HS) 
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The Educational Research and Development Unit (ERDU) of the Graduate School 
of Industrial Administration at Carnegie-Mellon University was formally established 
in January 1970 by a five year grant from the Ford Foundation* In July 1970 an 
additional two year grant was received from the IBM Corporation for work on a 
specific project within the general purview of ERDU. This report describes the 
research and development activities in which ERDU been engaged during 
January • December 1972. Prior reports (Klahr 1970, 1971) summarize the earlier 
activities • 

The principal activity during this year has been the further development 
and programming for the Educational Assembly System (EAS)« Although work has 
continued in the evaluation of student attitudes, faculty productivity, and 
teaching effectiveness, it has been proceeding at a much slower rate than 
previously. This report consists of two sections, a description of our progress 
on the EAS, and a brief summary of the status of the other evaluation projects. 

THE EDUCATIONAL ASSEMBLY SYSTEM 

I. Overview - - ^ 

• Our previous reports have described the overall purpose and scope of the 
EAS. In this section we present a summary in the form of questions and 
answers . 

A: What is the Educational Assembly System (EAS)? 

The EAS is a system that can generate information about educational 
materials organized in a curriculum best suited to each individual user. 

The potential user is a student with an educational goal ranging from 
something quite specific to a totally general (or ambiguous) goal. The Only 



constraint is that the goal should be achievable by completion of some 
sequence of educational materials, e.g., books, lectures, films, courses, 
seminars, tapes, articles, or problem-sets. Given such a goal* the EAS functions 
as would an enlightened educational consultant who had a vast awareness of 
most areas, subject material, job requirements, etc., and who had the time 
to serve the particular needs of this individual student. Such a consultant 
would be expected to suggest a program of actions tailored to the student, 
the completion of which would accomplish the given goal. 

A highly simplified sketch of the basic components is presented In 
Figure 1. The data base consists of two parts: a) structured descriptions 
o£ educational goals and materials (modules), and b) a network of word 
relationships (semantic net). Both of these are created by various subject- 
matter experts. The user inputs his goal, including information about the 
area, level, time he wants to spend, etc. Then the system interacts with 
questions about possible inconsistencies, prerequisites and other relevant 
information. The EAS programs attempt to "understand" (see Question c) 
the student's goal and then searches for modules that satisfy the goal and 
the side constraints! Further interactions may occur between student and 
system. Finally, the student is presented with the optimal curriculum. At 
this point, he can recycle at any desired level of detail, or he can leave 
the system and pursue his curriculum. (Note that the EAS does not retrieve 
the actual materials; it directs the student to them. In fact, it is more 
accurate to say that the product v;hich the system generates is a study guide , 
individualized to ct pcirwiculcir ^cal and student.) 
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B: How does it differ from a printed collection of curricula? Couldn't 
we just keep the various curricula in a file and let the student 
browse through them, choosing the one he wants? 

Since the system exists as a computer program^ we could indeed hand 
simulate the entire operation. Consider what it would require. Imagine that 
some area requires three subgoals to be accomplished before the stated goal 
can be considered to be satisfied. Tor instance^ a student might be expected 
to look into linear programming (LP), decision-theory, and applications in order 
to accomplish a goal of introductory operations research (OR). In turn, each 
of these subareas might involve three subareas. For example, applications might 
include inventory, scheduling, and forecasting. In turn these might each refer 
to three different pieces of work, other subdivisions, etc. Thus we have 40 
modules forming one specific curriculum in OR. Notice, however, that the subgoal 
of LP could have been a valid goal in its own right. Similarly, one might have 
wanted to pursue just the inventory applications in OR. Hence, any one of the 
various subtrees below any node is also a possible curriculum. In our 
example, there arc 40 possible subcurricula. In addition, since the 
•student might have any combination of prerequisites, any one of the sub- 
goals of any goal may be omitted in some particular case. In general, there 
are as many different curricula as there are actually different combinations 
of the various subbranches. That is, there are as many different curricula 
possible as there are subsets. Since the number of subsets possible from 
n items is 2^, we have a rather large upper bound (2^° in our example). In 
addition, 2^ docs not include permutations. Since sequencing is podagogically 
relevant in a curriculum, the upper bound is raised. Thus it Is clear 

that when one is talking about between 1,000 and 5,000 modules, the 
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conccpt of keeping a conventional file becomes, at best, clumsy. In 
addition, we then add the issue of updating, changing, and editing onto 
this large (and repetitive) universe to clinch the argument that computerized 
data management, and heuristic file handling is the only feasible approach. 
(However, this does assume that the criteria of individualization of study, 
and personalization of curricula holds. Needless to say, the status quo is 
one of uniform curriculum summary with limited and specified variation.) 

In addition to the sheer-size argument given, the EAS system performs 
other functions strictly orthogonal to a file's conventionally static 
conception. The original incoming goal of the student is analyzed and 
evaluated. In^n interaction with the student the EAS generates suggestions 
for improvement and allows responses by the student. Information gained 
during this period is used to focus and limit search. The system has the 
semantic capability of transforming a fuzzy or poorly posed goal into one of 
a well-defined search procedure in a hugh search space. Ambiguity is transformed 
into exponentially large possibilities, whose size is then pared (though not 
quite exponentially) by a heuristic search, relying on the trade-off of speed 
of the search to compensate for the size of the generated space. . 

This effort, in real terms, makes the concept of a file substitute inappropriate; 
it is much like arguing that a 100 by 100 matrix can be inverted by hand by 
certain efficient algorithms. It is true, but no one would be willing to 
undertake the task. 

C: How does the system .I'understand" the naive user? 

The system's understanding rests on its use of a semantic net (in 
conjunction with syntatic transformations) developed by the expert encoders. 
This 5cr.!antic net contains infonnatiOri about each terb uscd in the description 



-6- 

of any educational module. For example, one module may be described, in part 
by the term OR. The semantic net might hold the information that OR can be 
considered a subarea of applied mathematics as far as the use pf that 
module is concerned. In addition, the net holds the^in formation that LP is 
a subarea of OR (just as is the transportation-problem, etc.). Then, when 
the student asks for an area of applied mathematics that includes applications 
to transportation-problems, and vhen no modules are described thae ^yay, search 
operators on the net will derive the possible substitute of OR including 
applications to transportation problems. Presumably there is a module that 
has this description. Net operators are part of the system design which we 
believe will stimulate reasonable semantic inference in this task environment. 
Similarly, the student may have; phrased his inquiry askew to the way the 
module is described. As an example, the student may have been interested in 
•'OR or simplex-method." Thus syntactic transformation are also applied. The 
sys*:em will search for a module by this name (and probably not find one); 
however, it will also search for a module with the name "OR INCLUDING simplex- 
method." Both semantic and syntactic changes in the initial goal are made 
only when the student's original request cannot be directly satisfied. Such 
changes are also made for internally generated subgoals when such subgoals 
are subsequently found to be not directly satisfiable. 

D: What uses might an EAS have at a school such as GSIA? 

Use i: Intra-Course Supplement 

Assuming that the faculty member has supervised the creation of mr<'^iiles and 
a net corresponding to the particular course he is teaching, he may wish to offer 
Independent studies as part of the course (perhaps toward the second half 
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after introducing the basic material^ etcO^ He may wish to allow more 
motivated students to investigate a large number of related ^ adjacent, or 
©ore specialized areas with which he does not intend to concern the whole 
class. Such a supplementary capability could be available via the EAS 
system. It would act as an independent consultant in the course (or to offer 
another analogy, a knowledgeable Ph.D. stur.ent as your assistant in the 
course, though in this case, constantly available). 

Use ii: Full Course Use 

A faculty member may wish to give (or see supported) a particular area 
in his field of interest. However, he may not wish to. offer another course 
in that area (especially in addition to his regular load for the semester). 
The EAS system can accomodate such a desire. Since the system has as one 
of its central features a semantic capability in crder to deal with fuzzy, 
ill-posed, ambiguous, or poorly posed inquiries, it can handle students who 
wish to pursue some area but who have no real expertise in that area (for 
otherwise they might very well proceed entirely on their own with no 
assistance from anyone). The faculty member, by suitably supervising the 
net construction, can make such an independent studies course available. 
Such a course is given by and supervised by him; but it is largely unattended 
and requires little resource investment by him once the areas have been 
encoded. (Thus GSZk may move from an environment of repeated course 
production towards one of course management where professors manage the 
student's progress rather than regulate it.) 

Use tii ; Prerequisite Resolution 

GSIA already uses an informal subsystem to accomodate students who do 
not have certain prerequisites. There are video tapes on the use of TSS, 
FORTR.\N, etc. In addition we also rely on certain mathcsaatics courses 
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taught in the nathcmatics department for those who need or desire such 
foundations. This later resource is at times not optimal since there may 
be partial coverage of the material needed, or in other cases, overkill. 
The EAS system can accomodate the demand, for quite diverse needs for pre- 
requisite subjects that may support, impinge or intersect the particular 
faculty member's current course material. A properly created net allows 
the student to access a multitude of prerequisites at many stages (and at 
various levels, etc.) in subareas, as needed. Not only are such prerequisites 
made known (or made clear) to him, but the faculty is relieved of the burden 
of managing such diversions for each course. - As a consequence, fewer 
assumptions need be made about the student, and the staden^ need impose 
fewer constraints on his range of formal study. Moreover, the effort now 
directed in courses to establish prerequisites, which is often given limited 
time or resources (e.g., chapter 0 of the book), may be rechanneled elsewhere. 

Use iv: Course Design 

It is expected that as a faculty member develops a richer and more 
elaborate network (and as the system accesses other related networks), he 
then can use the system for course design. By entering the profile of the 
normative, hypothetical student he expects to teach, as well as the goal 
that represents the courses* subject area, he can use the curriculum 
generated by the system as the basis of his own course outline for that 
subject area. Since the system has access to not only that faculty member's 
net but other nets as well, the aggregated course production capacity of 
the faculty becomes a partially shared resource. In conjunction with use C, 
some of the more unrewarding parts of course generation (i.e. - prerequisite 
resolution) could be avoided. 
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Use v: Resou rce Evaluacion 

^ 

Again in the same context of a. multi-net environnient , the administration 

(or head of a department, etc.) may enter certain goals, representing areas 

or topics he wishes to see supported. Then the resulting curriculum 

becomes a resource evaluator. The more the system can pull together various 

parts of many (perhaps diverse) nets and complete the curriculum, the more 

the total educational system already has the resources necessary to support 

such a goal. The system is able to indicate the kinds of prerequisites 

and subgoals it was searching for but failed to find. Wliere the curriculum 

indicates missing portionis is where resources need to be directed. Thus 

a certain amount of inventory control is possible. 

Use vi : Generalized Prottram Support 

By generalizing the multi-net environment to its natural limit and 
hjrpothcsizing nets that cover all the areas with which some program is 
concerned, we can then use the system as the mainstay of the program itself* 
The student *s main task is to move through the net, extracting the curri- 
culum that best suits his goals and completing that curriculum* The whole 
program becomes defined by the system itself* For example, perhaps we 
wished to support a full political science program for those students 
wishing to include the classical areas of political science in addition to 
the subareas we already support. However, we may not wish to invest any labor 
in the project (i.e*, permanent faculty position). Then we would generate 
a net and collect the resources that were described in the net (e.g., books, 
filtis> courses at Pitt, journal articles). The net would act as the supplement 
program, making available suitable curricula, as appropriate , for a wide 
range of inquiry. With a minimal updating, a classical political science 
program could be made available, at a supplemental level* ^This use could 
of course be applied to the school or university level too. 
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E: Given thcit the EAS is supposed to do the job of a talented and knowledgeable 
human, wouldn't it be cheaper just to let humans do the job? 

It appears not (or can be made to appear not). At this point, we must 

make a host of assumptions because there are several educational environments 

in which the EAS might be utilized. Let ^ ^ ^ the EAS with a "conven- 

tional** approach in a specific application: the offering- of an individually 

s 

planned "major" in a single traditional area (e,g., psychology, math, etc.)/ 
Wc assume that a university wants to offer such a program to augment their 
existing program. A major goal of the program is the construction of highly 
'individualized curricula that are tailored to the specific needs and 
abilities of students. Such curricula are intended to be well structured 
and to indicate to the student the relations among the various prescribed 
instructional modules. 
1. Cost of EAS 

a. We assume that the structure of an EAS (i.e., the search 
programs, the goal language, etc.) exists, but that the content 
area (e.g., psychology) has not yet been encoded. Thus, our 
cost estimates will reflect the amortization of this initial 
investment in expensive labor. 

b. The mainstream of a discipline can be captured by encoding 
approximately 15,000 modules. This estimate is based upon 
our experience with the areas of artificial intelligence and 
management science. The appropriate encoding of modules is. 
central to the success of EAS, and requires the skill of 
top level subject-matter experts. We estimate a total cost 
of $100,000 to encode a single area as follows: 



1,000 modules from top experts: 



$ 15,000 



4,000 modules from major researchers: 



35,000 



10^000 modules from graduate students: 



30,000 



$ 80,000 



Keypunch, tape, computer time (for data entry) 



20,000 



$ 



100,000 



The annual cost of storing and maintaining the data base 

for these 15,000 modules is estimated at $20,000. This is 

a fixed cost, independent of actual usage, based upon existing 

storage charges at CMU, assuming the 15,000 modules require 15 

million characters of storage, plus some updating. 

The incremental cost of using the system is based upon the 

number and duration of system-user encounters. Assume that each 

student using the system has a session tc define his initial 

curriculum, and another three during the secaester to refine 

and update his curriculum. Assume that each session lasts 

about 15 minutes and that connect and processing costs are 

$5.00 for each session. That yields $20 per student per 

semester* Assuming the student uses the system again during 

the second semester, we get $40 per student, per year. 

We estimate aggregate costs by assuming tvo different usages 

and accounting procedures. 

1. Assume a small number of users (departm(£int) and pro* 
rate the fixed storage costs across all users« For 500 
students we have; 
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storage costs: $ 20,000 

usage: $40 x 500 $ 20,000 

$ 40,000 

$80 per student per year. 

il» Assume there is a large number of users and amortize 

encoding costs over 2 years. For 2,000 students we would 
have: 

amortized encoding costs (2 year period) $ 50,000 

storage 20,000 

usage: $40 x 2,000 80,000 

150,000 

i 

$75 per student per year (including 2-year 
amortization) 

Cost of Conventional Approaches 

a* Consider the labor Input of consulting with 500 students so 

that they can create and follow individualized curricula of 'the^ 
kind produced by EAS. We assume that student- faculty inter- 
action takes somewhat longer than the EAS interaction. Thus, 
each student consumes about 1 hour of faculty time (in each of 
four sessions) per semester, or 1 man-day per year per student* 

b. Assuming $20,000 per year for faculty member, including fringe, 

overhead, etc*, and 200 working days per year, gives us the cost 

of that one man -day: 

$100 per student per year 
for "consulting" 
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c« Another way to pose this is to estimate the cost of acquiring 
a faculty that can devote 500 man-days to consultation and 
curriculum planning. Once again^ assuming 200 working days per 
year, we need 2 1/2 man years at $20,000 to meet the demands of 
500 students. We require 2 1/2 man years x 20,000 or $50,000 
per year, or $100 per student per year. 

d« There is no reason to asstime any economies of scale In the 
tnan-to-man approach, so the per-student costs would be the 
same for 2,000 students as for 500. 

4 

3g Comparison 

We can, summarize the annual per student cost as follows: 



Students 
N 


conventional 


EAS 

without amortizing 
encoding 


EAS 

with amortizing 
encoding 








2 years 5 years 


^ 500 


100 


$80 




2,000 
10,000 


100 
100 




$75 $60* 
$47* $A4* 



based upon calculations similar to Section l.e.ii 
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II. Creating the Data Base 

As indicated in Figure I, the EAS requires a data base consisting of 
descriptions of educational resources that can be searched in a meaningful 
way when the system is attempting to design a curriculum for a user. In 
this section v;e will describe the procedure we have devised to collect this 
crucial information from subject-matter experts. 

A: Area definition 

The first step in creating the data base requires decisions about the 
general size and scope of the knowledge domains in , which the system will 
operate. For our prototype EAS, we have decided to work in the area of 
management science (MS). The EAS should be able to construct a reasonable 
curriculum for a range of students whose educational goals vary from introductory 
to advanced MS with an available time span of approximately 5 to 25 months. 
In Figure 2 we have listed some approximations of the number of modules this 
will require, as well as a rough mapping into more convent ionai academic units. 
We estimate Chat appfdximately fifteen thousand moddl^ will be sufficient to 
span the areas of knowledge under the general heading of MS. 

B: Module collection 

These modules must be created and encoded before they can be entered 
Into the system. Our collection strategy is to get a small number of top 
level modules encoded by subject matter experts (e.g., GSIA faculty). 

Top level modules usually describe the accomplishments of an educational 
goal in terms of the completion of some sub -goals. We call these structure 
modules. Ultimately an educational i>oal is described in terms of some actual 
activity that must take place (reading, problem solving, etc.). We call these 



A^ tynes of r.odules 
Area 



ir.ana^" en enc 
science 




Level 



introductory 



intermediate 



advanced 



Time (academic yr,) Equiva'e 

in Tiir. 

coo C 60 

►5 months .(-^ yr, ) 9 cou 



'10 months (1 yr, )- 



^15 months (3.$ yr..) 
^20 months (2 yr. ) - 
25 months (2$: yr. ) 



.18 " 
27 

.36 " 

-45 



S» number of modules 

approximately 7 topics per course 

3 sub-topics per topic 
3 s ub"Sub-topics T>er sub-topic 
63 modules per course 

x5 variations of each module 



total of 31^ modules r>er course 

Assunin? 45 courses, we set l4 >17 5 modules. 

inear al^^ebra course 



- /in addition, assuming- tv/o calculus and one li 
\as additional suppleiaent, v/e ad d ^^^5 modules. 

^^.----^tal modules anticipated j IS « 120 modules ■ 



) 



Figure 2 

Size of Data Base for Management Science 
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content modules. Our module collection strategy is to have top-level people 
devote most of their effort to structure modules - thus defining the general 
structure of knowledge in their areas* Content modules, and lower level 
structure modules will be encoded by Ph.D. students, since they are often 
more familiar with appropriate instructional materials for relatively well- 
defined and narrow goals. 

During this year a substantial effort was devoted to the construction of 
a set of guides and instructions for the encoders. They are contained in 
Appendices A, B and C. These encoder guides were then used by several top-, 
level and second-level subject matter experts to encode the first set of 
modules to be used in debugging the EAS programs. Rather than repeat the 
substance of the Guides here in the narrative, we suggest that they be read at 
this point in order to understand the encoder's job. 

Several GSIA faculty and Ph.D. students worked as encoders, producing a 
small core of modules (250) and giving up some data for estimating the labor 
jiosts of module creation, as well as suggestions for revisions in the 
encoding guides. 

III. Programming the EAS 
A: Status 

As of December 1971, the general system design had been completed. As of 
December 1972, the entire prototype system has been implented and is running 
on the IBM 360/67 at CMU. The entire system, both the processes and the data 
base, is written in LISP, a recursive function language, in an interpretive mode. 
This has provided us with a very flexible and powerful language for the initial 
impler-icntntion of our ideas. However, it has been rather inefficient in the 
use of computer time and space, and like most developing systems, it has 
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many "patches" throughout. Our next step is to totally redesign the "pure 
programming" aspects of the system so that we can increase the usable data 
base and decreasfe the running time. 



B: Examples of the EAS in Operation 

In the first test to be discussed, we pose the goal of asking for instruction 
in the area of "quantitative methods including linear optimization." [The special 
word "including" is used in the system to mean "with particular emphasis on the 
sub-area of"]. This goal was posed at the introductory level, with a mastery desired 
of the equivalent of making a C, with a motivation level estimated at 6-8, and time 
desired of 4 1/2 weeks, using any media. Such information is usually abbreviated 
by the system as : 

.ft ti 
quantitative methods including linear optimization/ intro. , C, 6-8, 4 1/2 wks. 

Note that media is dropped if no preference is specified. 

As part of the first use of the semantic net, which stores the combined collection 

of cognitive maps of the various experts who have coded modules for the system, the 

system prompts the student for prerequisites that may be pertinent with respect 

to this goal. In this test, the areas prompted for included: 

linear optimization 

-quantitative methods 

dual solutions, dual problems, duality 

linear programming 

objective functions 

constraints 

initial solutions 

change-of -basis 

sensitivity 

simplex-method 



Given the above goal, there arc no such modules described this way or which even 
use some of these terms. Using syntactic expansions of the goal together with se- 
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m-antic relaxations (which will be explained below) the program considered 

six possible choices of modules that might satisfy this goal. These included those 

with the area parts given by: 

linear programming including prime solutions 
linear programming including problem formulation 
linear programming including geometrical solutions 
linear programming including simplex-method 
linear programming dual solutions 
linear programming 

Side-stepping the five erroneous ones (which focused on some particular aspect 
of linear programming rather than treating it in general as was intended by 
the goal), the program properly chose the module "linear programming", intro- 
ductory, etc. one might note that it also was not confused with a module within 
the current universe described by "operations-research" even though this is another 
virtual synonym with the term "quantitative methods" which does occur in the 
goal statement. However, the system discovers that the available module is 
for a period of time much less than the desired goal's time; on this point, the 
module is rejected, and hence a time failure is noted. Since this is the only 
related module the system finds suitable (area-wise), it later would be reported 
as the best try though deficient in time. Again we note also what the system 
did not do. Since the time was out of range, the system allowed a small varia- 
tion to see if that would be sufficient to make this goal fit. After this 
relaxation, and its failure, the system" checked to see if the evaluation of 
the goal suggested that the original time request might have been a (suspected) 
poor choice. The system finds that no flags wer^ set in the evaluation of the 
goal vis a vis time. Hence, failure is reported. In other cases, either small 
variations in the time permit a match, or goal evaluation has made us suspect, 
and larger time variations are allowed if trouble occurs on "time". 

in the next test, the same goal is inputted, but this time, the time is 
altered. As will be seen, we shift to 1 1/4 days. The goal is quantitative 
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methods including linear optimization/introduction, 1, 1, 3-5, 1 1/4 days. For 
this similar goal a module is found that is satisfactory; it is given by 

Linear programirdng/introduction, 1, 3-5, 1 day. 
In turn this goal has several "subgoals , which expand into sub-subgoals« These 
are given in Figure 3. Several points can be observed in the expansion. 
First, when a module was assigned, and then later found appropriate again, the 
system reassigned it rather than assigning yet another module. When the system could 
find no module that sufficed to fr tf ill subgoajs, it created dummy modules, in effect 
indicating to the student the necessary accomplishments he would have to achieve. In 
this particular test, one subgoal differs with respect to level from another; other- 
wise it is identical. Again:we note that the system did not falsely relax the level, 
thereby using one or the other module in both places. In general, subgoal modules are 
treated differently than the top-level goal, relaxation being one such variation. 

The next test re-enters this same goal once again, but in this case, the student 
indicates a prerequisite of 

linear algebra/intro. , 4, 6,6 weeks 
We note that a subgoal of "linear programming including problem- formal at ion" included 
the subgoal: 

linear algebra or matrices/intro. , 2-3, 4-9, 1 week. 
The program determines that the prerequisite will suffice (in fact the prerequisite 
claimed is much more than enough-— a fact that does not confound the evaluation), and 
so assigns the claimed prerequisite, producing the tiee that was given in Figure 3 
with the exception that we have modules for linear algebra, as shown in Figure 4a and b 

In the next tci.ts, we have a desired goal of the form'^linear programming or 
operations research'; Including simplex method," where the associated level desired is 
advanced, with mastery of C nlgcbrn ^ a motivation level estimated to be 7, and time to be 
invested of 6 weeks. This particular goal points up the capability of the system to 
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target goal: quantitative methods including linear optimization 
introductory, i; 3-5, 1 1/4 days 




accepted goal: linear programming/intro. ,1, 3-5, 1 day 
module: 



target 

suLgoal: linear optimization or primal simplex method/intro. ,2,4-6, l/4-l/4da. 

accepted: linear progranming including simplex method /intro. , 1,2-6, 2-4hr., 
module 



target sub-sub-goal: linear programming including problem- 
formulation/intro. ,1,2-6,1-2 hours 

accepted module: linear programming including problem- 
(#3) formulation/intro. ,1,2-6,1-2 hours 



target sub-sub-goal: linear algebra or matrices/intro. , 
2-3,4-9,1-1 1/2 weeks 

no accepted module; dummy module assigned directing the 
student to: linear algebra or matrices/intro. , 

(^M) 2-3,4-9,1-1 1/2 weeks 



target sub-sub-goal: linear algebra or matrices/intro., 

2-3,4-9, 1-1 1/2 weeks 
assigned module #4 



target module: linear programming including duality/intro. ,2,5-7, 1/4 - 1/2 day 

accepted: linear programming including dual solutions/intro. . 

module(f/5) 1,3-5,3-5 hours 

'target subgoal: L,P. including problem formulatiun/intro 

1,2-6, 1-2 hours 
assigned module #3 to fulfill this 

target subgoal: linear algebra or matrices, 

intro,, 2-3,4-9, 1-1 1/2 weeks 
assigned module #4 

target subgoal: linear programming including simplex 
method/intermediate, 1,2-6,2-4 hours 

no acceptable module found due to level; 
assigned module #6 




Figure 3 
An expansion of the goal 
linear programming/intro. , 1 ,3-5 , 1 day 
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f4 



\ assipnea 
1 profile 
I rfiCdule 



target: Linear algebra or matrices 

intrc. ,1,3-5. 1 ciay | 

acceiDted profile entry; linear algebral 
intro.,4,6, 6 weeks j 




Figure 4a 

Expnnsion of Goal Using Accepted Profile Entry 



handle very complex area-parts to the statements as well perform quite extensive 

and elaborate syntactical and semantic transformations on such goals. Some of 

the more obvious transformations included: 

(operations research including linear programming) 
including simplex method 

(operations research or linear programming) 
including simplex method 

operations research including simplex method 

linear programming including simplex method 

simplex method or linear programming 

• 

A more complete example of syntactic expansion is given in Figure 5. In 

addition to syntactic transformations, the crucial semantic alterations are formed 

some simple ones include: 

linear programming including prime solutions 

(quantitative analysis including linear optimization) 
including primal simplex method 

In this particular test, the program converged to two modules, described by the 
areas ''linear programming including prime solutions" and Vlinear programming 
including simplex method**. Since these two candidates both had a level of 
**introductory** versus the desired **advanced**, and since there was not sufficient 
reason to relax the desired goal to the above goals, the program correctly termin- 
ates with a description of its failure to find a module with proper **level**. 
In the next test, the desired goal was similar, but with a reduced level, so 
that n **hit** could be expected - and a reduced time to make the goal well-posed 
(reducing; the number of flags that might be posted). The system again focuses 



Figure 5 

Expansion of (LP or Oil) including iUnplex Method 
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on these two modules, choosing the obviously best choice of th'» two, "linear 
programming including simplex method*'. It proceeds to create the rest of the 
curriculum as given in Figure 6, 

In another test, the system considered the general goal .-irea of "operations 
research", generating the subgoals as given in Figure 7. 

Other search procedures are in evidence in these tests, though it would 
take a series of tests in which the desired goals and the module universe dif- 
fered by some slight variations ,in certain parameters to cause these differences 
to appear. For instance the system chooses those modules whose subgoals appear 
satisfiable by the system over modules whose subgoals (or a smaller percent of 
whose subgoals) do not seem satisfiable. In addition, each parameter of mastery, 
motive and media is optimized against, all other things being equal, in addition 
to handling the complicated cases where some of each of the parameters are 
satisfied to varying degrees. This search goes on in conjunction with relaxa- 
tion of parameters if goal analysis prompted us to anticipate trouble on some 
particular parameter. Finally, the system makes discriminations along ''context", 
such a context being built up from previously assigned modules as well as other 
information collected during the goal-input phase. Such information is requested 
on a "need to know" basis, where a heuristic recipe estimates the amount of in- 
formation that will be requested. Such a recipe number (corresponding to levels 
of inquiry) based on the inputted goal, etc., is presently operational and is 
calculated by the system. 

Even without the complete outputs of all these tests however, the few tests 
described above confirm the feasibility of the design proposed for an education 
assembly system for student-executed educational design. In addition, the feas- 
ibility of encoding and collecting cognitive maps has been shown. As an example 
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to^^-level tar^-et >^oal :{ linear r)ro£:rar;:p.insr or operations research) 

including: sir^plex niethod/ijitrcductory ,2 ,7 , , day 



assi^'r.ed v:odjle -1: linear r^rof:i:ra..T:incr including siruplex method/ 

introductory ,1 ,2-c , ^"g- day 




sut>^>:oal: linear programming including problen-f orrrulation/ 
intro. ,1,2-", 1-2'hours 



^accepted module 4'2 : linear programming including problem. 
\ for^nulation/intro. ,1,2-6,1-2 hours 



sub-subsoal; linear algebra or matrices/ii.tro . , 
2-3, ^-9 f weeks 



no niodules acceptable; : assign du-^^my nodulo ..-3 



S'-h^ir-.l: ] r ebr?./i:itorductory . ,2-3 ,^-9 t Isv/ee! 

?.cce'pted r.M^dule •r3 




Figure 6 

.0:! coj'-ii:vel fi'oal 



-ERIC 
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• •.•■i:-. /1, 1. ,3-3.1 day 



r--v., /t-'.-.lr :::?r':):':i t'r:-e-r,oV.:-clons/l, 3-5. 1/3-2/3 day 



J, ^ . 



^i;irr3ar-r2-o.-ra^.vin:V I:';i;:d::': dual-solv-cions/l,!, 3-5 » 1/3-2/3 day 
\ transnortation-problern/l , 1 , 3-5 fi' day 

^ transrortaxion--Drcolet^ I* GLUDI'riCr prime-solutions/1,1, 3-5 » i day 
trans-ocrtation-problem i;%CLUDIuG dual-solutions/l,l,3-5»'4- day 



\ \ critical-i^ath-r:ethQd/l , 1 , ^-5 » 1 day 

\ V ^•/Criticai-path-method IfiCLUDINGv /l,l, 3-5. 1/3-2/3 clay 

\ \ \ \ ^problem-f crmulauion ' 

\\ \ , critical-rath-method INCLUDING^ /l, 1,3-5 1 1/3-2/3 day 
\ \ \ ^ snl vitinn-f ormulation 



. \ \ solution-formulation 
\ \ dec5sT'^r}-theorv/ltl.3-5fl day 



decision-theory INCLUDIKG problem-f ormulation/l,l, 3-5 1 1/3-2/3 da;. 

1\\ decision- theory INCLUDING solution-interpretation/l,l,3-5tl^-2/^ d. ; 

1 \ \ , \ 

\ \ forecastins:-niodels/l,l,3-5t 1 day . \ 

\ \ --^ -■ . \ 

\ \ \ forecasting-nodels INCLUDING problem-f ormulation/l,lf 3-5il5-?^ ^2ay 
^forecastinv<2:-models INCLUDING solution-techniques/l,l,3-5fX/3-2/^cay 



\ \ scheduling:-r>ro blems/l > 1 , 3-5 1 p day ' " I 

\ f^K^ " ~ I 

\ \^ \^ scheduling-problens INCLUDING problem-f ormulation/l ,1 ,3-5.i day 

1 ^ schedulinfi-probleiiis INCLUDING solution-ideas/l ,1 , 3-5 1 t day 



\ inventory-problems/l > 1 > 3-5 1 1 day 

\ ^*\^ inventory-problems INCLUDING problem-f ormulation/l, It 3-5 1 1/3-2/3 d. 

\nventory-probl9Tns INCLUDING solution-interpretation^l,3-5iV5-4^3 ^ 

. i 
oiveratioriS-research INCrjDi:;G aiDr>lications/l,l,3-5.2 days i 

\x 



o - <^ o- - 



Ficrurc 7 
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of a more elaborate curriculum available from encoded modules (not all of which 
have been entered into the system, giving consideration to space, cost, etc) 
see Figures 8 and 9« 



computer- science/ 1 
2-4, 5-8, 3 months 




"Dgrarrining 
2-3, 6-8, 4-6 weeks 

2neral- programming/ 1 , 
2-3, 6-8, 3-4 weeks 



\ \prog. INC. 



^prog. INC. languages/1, 
2-4, 5-7, 1 1/2 weeks 

^prog. INC. implementation/l , 
2-4, 4-7, 1 week 



iprogran design /I , * 
3-4, 5-7, 1 1/2 weeks I 



heuris t ic- pr ogranming/ 1 
^-3, 5-8,^ 2-3 weeks 

\\heuristic-prog. INCmethods/l 
2-4, 4-8, 11/2-2 weeks 

lieuristic-prog. INC. heuristics/1 
2-4i 4-8, 1/2 - 1 week 



^Operat ions-Research 
s2-3, 6-8, 3-4 weeks 



>imulation/l , 
>-3, 5-8, 1/2 week 

^simul, INC, discrete- 
events-modols/1 , 
2-3, 4-8, 1 day 

► simulation- 
languages/ 1 J 
3-4, 5-8, .1 1/2 days 

iprobability AND 
statibtics/1 
^2-4, 6-8, l^week 

*prob, INC. density- 
functions /I, 
3-4, 6-9, 1 1/2-2 days 

^prob. INC. distribution 
functions/^ 
3-4, 6-9, 1 1/2-2 days 

iprob. INC. moments/1, 
3-4, 6-9, 1 1/2-2 days 
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♦ Logic/1, 

k 2-3, 6-8,' 3-4 weeks 

\propositional- 
calculus/1, 
3-4, 6-9, 1 week 

\ 

content module/1, 
2, 4-8, 2/3- 
1 1/2 weeks 



\ 



mathematics /I, 
1, 1/3, 1 week 

predicate-calculus/1 , 
^2-4, 6-9, 2-3 weeks 

^content module/I, 
\2-, 5-7, 2-3 weeks 

^prepositional 
calculus/1, 
1,1-4, 2/3 
2/3 week 




''optimization/I, 1-3, 4-9, 1/2 week 



ppti. AND derivatives /I , 
3-4, 5-8, 1 day 

bpti. AND (minima AND 
maxima) / I , 
2-4, 5-8, 1/2-1 day 

opti. AND Lagrangian/1, 
2-4, 6-9, 1 day 



linear- prograrjning/l ,2-4,6-8,1 1/2 weeks 



1 inear- nrograrjning- problem/ 1 

1- J, 4I7, 1/2 week 

sin*plor<-nietiiod/l , 

2- ^, 5-6, l/:-2/3 week 

dual ity- tlu'or/ ^ 1 , 
2-4, 5-S, l/'3-l/2 vock 
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EVALUATION PROJECTS 

The ultimate goal of all of ERDU's projects is to increase educational 
efficiency. In the case of the work on the EAS, we have taken the route of 
actual construction of a system that might contribute to such a goal in 
appropriate educational environments. Other ERDU projects have been directed 
toward the efficiency goal through different paths. 

Our explorations of the use of television technology have resulted in a 
preliminary survey in the Pittsburgh region of the demand for a "televised 
masters degree" jointly offered by GSIA and Carnegie Institute of Technology. 
Preliminary results seem to indicate that there is not enough local demand 
to warrant our pursuing this any further. Our in-house use of videotape 
equipment has increased to the point where almost all of our students have been 
involved in using the medium in some way for educational purposes. However, we 
now believe that no major break-throughs will result from this sort of 
technology within the conventional academic structure. 

The study of departmental productivity has lead to some preliminary 
results and has suggested some system changes. The investigation of the 
sensitivity of research productivity indices to various assumptionsabout 
aggregation procedures has demonstrated that any index that is based upon a 
reasonable set of assumptions will correlate highly with most others. In 
particular, an index that simply counts total number of papers published ranks 
faculty in almost the same order as one that corrects for rate, quality of 
journal and pages. 

The general study, of which research productivity was only one component, 
also required measures of teaching quality, entering student quality and 
final student quality. During the course of the study it became evident that 
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CMU did not maintain the records (at least in any accessible form) that would 
facilitate the investigation of questions of this sort. The University 
Administration is now designing a new n^nagement information system that will 
keep track of not only the costs of running the university but also the 
quality of its products: research and education. 
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PARTICIPANTS 

The principal participants in ERDU this year were Professor D* Klahr, 
Director, S. Evans, Research Associate, and Dermody, Senior Systems 
Programmer o Professors C* Kriebel, G. Thompson and R. Weil assisted in module 
encoding as did several of our Ph.D. students. J. Bloom supervised a small 
staff of Masters students in the use of our video tape facility. 



Appendix A 

MODULK KXCODKR'S GU TDK: DnSCRim^ION OF YOUR TASK 



Wo wish to enlist ycnr oxpertisc in the creation of a data base for 
-an Educational Assembly System (fiAS) • In this Encoder's Guide we will first 
describe a few features of the design and operation of the EAS. Then we will 
describe in detail the task we have set for you and the procedures we would 
like you to follow. 
System Design and Operation 

The purpose of the EAS is to generate, upon demand, highly individualized 
curricula for a wide range of students with diverse backgrounds, characteristics 
and educational objectives. The ultimate design goal is to simulate an omniscient, 
perceptive and indefatigable human educational consultant and curriculum designer. 
The intended user of the system, a student, describes his objectives in terms of 
® gQ^l statement. Then the system selects - from a large collection of 
previously encoded educational resources - a sequence of educational modules. 
These modules are descriptions of educational subgoals whose achievement will 
satisfy the student's goal. In most cases, the subgoals have their own sub- 
subgoals, etc. The higher level subgoals are often described using structu re 
modules, which lay out the relationships between the broad areas of knowledge 
that are related to the student's stated goa*. The "low level" goals are more 
often achieved through content ir.odules; they correspond to specific educational 
activities (reading parts of books, solving problem sets, taking courses, etc). 
We slu-^ll rjcurn to a fuller doscriptioii of both structure and content models in 
a later section. 

<ysc.;: r.' ;i. ir s suv-ral kirJt; oL ir. for*. union. First it nwods to 
knw i\bo\.t .••.tti:io;:t: : his c:oals, his backi^round, his preferences, etc. Such 
Intorr I is su-^IicJ .liroccly by t!\o InJividual ut;er dtirins his initial int.r- 
..il'. • r.\..ti if will njL vO.v* rn v lurtiior rure. ^-.'cond, it \u .'kU 

to tw.'w K\z the -•rrK:-..rt* of kiuv,;:*^^* in the jriws for which it i.> crcntin\: a 
currictihi'\ Tor o>M".^;)Io, It noods to kno\; thnt chc» uron of linear proi^rammlnc, has. 



A- 2 

as subarcas, material on objective functions, on the simplex method, etc.. 
Part of your Cask as an encoder is to precisely specify your view of these 
relationships or subgoals in your area of expertise. This Guide will indicate 
how you can do this in a manner that is intelligible to the EAS through the encoding 
of modules. 

Another kind of information needed by the system pertains to the "meaning" 
of the terms used by you, by other expert encoders, and by the (typically naive') 
student. For example, it needs to know that (for some experts) "operations 
research" is roughly synononous with "quantitative methods", that it is a subset 
of "applied math" and "management science", and that "dynamic programming", 
"linear programming", "statistics", and "queuing theory" are all subsets of it. 
You will be asked to supply this information for each of the descriptive terms you 
use. The system will then construct a semantic net by building a file of all such 
relationships provided by you and the other encoders. The distinction between sub- 
goals and the semantic net will become evident as you begin to actually encode modules. 
Your task: overview 

As indicated above, the student communicates with the system by specifying, 
among other things, his educational goal. The system is designed to use goal speci- 
fications in another way: in the description of modules. Both content modules and 
structure modules are encoded in terms of the goals that they satisfy. Your task, 
is to (1) create and encode a module that will satisfy a high-level goal [specified 
by the system designers] and (2) to encode the subgoals generated [by you] in doing 
(1^. This .process rny concinuo for several lo.vols. In order to do (1^ and (2> , 
you need to knv^/ more about the language in whicli goals are to be written and about 
the other inionnation reouirod to encode a module. 

Com stj; . ' \t> 

To v:.::.t.- I..C tuL^k cK.ir, vo :;hall briefly dv.'!:cvil'c a i;oal stato-.v.ciK. O'oro 
detail uDouL ^oals will provided laccD. A goai Uaa live pares: 

(1) The iiVCA givfs the subject or knowledj*e duuain tliat the ntodule considers 
"ci^-.*., i: t. r Si'ii Mco" or "ov*vVu>:"Ic!;"> . 



Tcrn,s n.y be ccnbinod with the operators AND. OR, liXCLUDC^G, and INCLUDING (to be AO 
explained shortly'*. Parcntho-ses should be used to avoid ambiguity. Some examples 
of area expressions are ..(cor.nn>tcr..c<oncc INCLUDING progrn:nning) AND (mathematical- 
logic)" , "microeconomics OR (macroeconomics KCLUDirJG public-oolicy-ccononic)." . 
"psychology CXCI.fDlNG (math AND statistics)", etc. 

The last n;o goals use the terms INCLUDKG and EN'CLUDING.' The system differentiates 
these two terms from AIs'D and NOT by its actions in the search for suitable modules.The goal 
pf "psychology nXCLUDIXG (math AND statistics V requests or strongly prefers psychology 
with no math or statistics. If there were only one possible module to choose, and it 
included some "math AND statistics", then the search algorithm would "relax" the 
constraints sufficiently to allow this module to suffice, for this case. Thus , 
EXCLUDING is a weak form of NOT- Similarly, INCLUDING is a weak form 
of AND. "Psychology INCLUDING math" similarly requests the two, but the system may 
relax its search, if it is unable to meet the added demand of math. Observe that 

Avn ^.^h•• requests both areas in a module. This is not their se.'vurate 
••psychology AND math ^ 

anion but rather nn intersection or integration of the rwo. 

Note the lack of verbs, prepositions, and' mndl'Siers. The area part of the 
goal is not a description. Rather it is an indicator of the subject material's name. 
Hence, one must convert such phrases as "wiring of the machine" to "machine-wiring". 
"Setting up L. P. programs" is converted to "L.-P. INCLUDING formulation". (If 
necessary, plirases are to be converted tq new hyphenated forms as "making of transistors" 
becomes "transistor-making"). In Figure 1, we have listed appropriate' area encodings 
for several topics. 

(2) Aic level spccLiios cno. oi." throe Levels ol treatment; introtiuctory , 
Intcrrcdiatj, or advanced. Tims, wo may have "economics/ introductory" or "progra::'.n:ini; 
ASO co;.,^wL. :• i<.:\<k- j.J . ..p.. . o" . .'.lo 1. i:ivcu ..lUi" c.u "/', r.'.ocii iv .vvv: t:-.:;".J. 

to tho h "'". 

■ -•■■•'1 .•iv'.-' .m ••-.-t: i! '.uto of tho lovd of ^K-to-;-'! i s'-.-.-.mic o:<:'« .-Ci-d 

a.<-M«ci.'t.'.'. vl:''. tl'v <U-^i!<" t»' -tLt-.'U .:i!v.-.:uo op.o':! •.•..".uTal Kno-.-loJ. in t!ie aro.i.' 
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Tonic 



Solne Examples of Encoding 

Arcn-onrf of COAL 



•'the art of progrrinuiing** 

"learnirg to program" 

"ha.v" to evaluate a canned pi-ogra-^i 
with respect to suitability of needs" 

"functions of several variables" 

"management science applications 
of linear prbgranming" 



"mathematical approaches to 
decision-making in management 

"use of Lagrangian multipliers" 



"programming robots" 



programming 
programming 

canned-routines INCLUDING evaluation 

complex - variables 

linear- programming INCLUDIKG 
management- science-applications 

or 

management- science- applications 
INCLUDING linear- programming 

(depending on intendciJ emphasis) 

management-science INCLUDING 
(mathematics AND decision-making) 

(optimization- techniques INCLUDING 
Lagrangian-multipliers) INCLUDING 
examples 

complex- information-processing 
INCLUDING robots 



or 



artificial-intelligence INCLUDING 
robots 

(depending on intent of encoder) 



"describing decision making" 



probability - theory INCLUDDCG decision- 
trees 



"^.':.r iv:." .:n.u-i" ; 



or 

d o i :u or! - a • ) :•, 1 y s i s I N'C LV D ING 
d f c i s i on- i 1 ov - d i ii g r ;:in 
(dcpi vdi:\ \ on utccr.t^ 

i n V on i or\ - o on c r o 1 

or 

1 in\ n t .-^ r V - o oil t r o I - 1 0 0 ! m L c u o i> 
oa intone » 
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This is roughly uquivalcnL to a pass in a pass/fail course. Level C is ono of 

minin.al satisfactory prcficiincy. It asran.ics the studont can work v;ith or manipulate 

the material, ideas, etc., or in our analogy, made a C in the course or module. 

Level B suggobts that the student is under full control of the area when done with 

the module* He can fully manipulate and handle the material, doin^ the bulk of all 

typical exercises, ansv/cring or responding correctly to most of the issues, etc. 

This corresponds to a B in a course. Level A, the highest level, suggests the student 

is ••master'' of the material. This includes the ability to study or advance by himself 

or to handle ncv? r.iaterial in the area. This is analogous to making an A in a ccrjirse. 

(4^ The notive|rcpreser.ts a estimate of the appropriate degree of seriousness 

of a student who r.ight use this module. This is given as a range (between 1 and 9 with 
1 the lowest. 9 the highest). An example might be "1-3" which suggests a somewhat lo.. 
motivation in pursuing this area. The range 8-9 suggests a vary high degree of 
seriousness in pursuit of. this module. A module with 1-9 indicates that the module 
is appropriate for any student, regardless of his motivation in pursuing th'at subject 
matter. 

(5) Finally, the last part of the goal is an estimate of the amount of time 
alWed for completion of the module. This estimate may be given as a range, in 
any units appropriate, assuming the student will spend full-time pursuing the goal, 
exclusively. As an example, consider the full goal: progrOT.-..ing/ introductory, C-B 
7-9, 2 months.. This indicates that the student wishes to learn programming at an in- 
froductor>' level, requires that such material impart' a mastery level between C and B, 
and alleys a total time of t\70 -nonths for co.upletion. As an encoder, ifVou were given 
such a goal, .our job voalJ b.i to spociiy oJ-ic...tio.^..-il r-Uivitios ^or Ci.o studoat thnt 
vould sntisi>- tiiis joal. 

Sc I''.'. V Lvi i'l.' >. ■ L -■ c ■ ■ '.. '.i ■ ' *■ 

Given a ^a^i^nt 's .;o.n , f.i.^ n.viv.lo':; puriH^so is to i-nnMo the student to fulfill 
his 5-.oal. ..ich r.o..ul. r. -..r.^o.-nt.; .1 po.= si>U: j^cal th.-.t a -stud.nt r.iv have, 

aud to fju-cjr rcCuli: lo >.t.>.'oJ.o a (;>o.-.-; LbU- ' i;oai . 

^:.rLV.*.--..- - ^•■i-' - : :.h o.:..-. :'::iu-;:l..r 

L'>ourciM; aro r.iri.v.-vi t.. .s.aisiy the -oal. llius tho "piM-oqu iciit.s" section of tU. 



roourc 
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Structure nodu l (2S are often associated with more general gonls that are in turn depen- 
dent upon gcnern] subdonains. In this case, goal satisfaction does not require that 
resources be retrieved directly; instead there is the need to specify subgoals, whose 
satisfaction will permit the accor»plishr.ent of the original goal. 

To exemplify these two types of modules, consider two goals: (1) ''I want to learn 
about • theory of the firm* at an introductory level'' and (2) ''I want to learn about 
'management science* at an introductory level''. If we wish to encode modules to 
satisfy these two goals, in the first case we may encode the book by the same name. 
The mastery of the book wij^.l he the suitable action to fulfill the goal. Prerequisites 
to reading the book might be an awareness of basic economic issues plus some mathematical 
aptitude. This is a content module. In the second case, we might structure the 
field of management science as consisting of the subdomains of operations research, 
economics, and industrial administration. Having done this, we note that the mastery 
of these subareas at an introductory level is a suitable action whole accomplishment 
will result in the original goal .being fulfilled. Hence, the goal of understanding 
the material in this module implies fulfilling the subgoals of ''learning introductory 
operations research, economics, and industrial administration". There is no material , 
per se, that is to be retrieved. This is a structure module.' 

One might hypothesize that any particular module may be in fact a "mixed case", 
part content and part structure. This is not the, case, hcr.;ever; the mixed case can 
be decomposed as follows. Assume that an expert wishes to encode the goal ''introduction 
to operations research". He may feel that the student should read the first two chapters 
of Aroncfsky's boo'c, the iiv^z hall oi ?olIrr*.s ;-ook on pre*- ability and cot an introductory 
undor:>t jndinv; oi sir.uiation, linear prograr;.r.;in^, and statistics. This sequence oc accions, 
wlii,:!^ vould :;:!fill tl.o .^'uM:Ld o;:ly as a t^tC of zc> 

bo ful:ilKd. '\\\v.$ i,y: c!ic jmL ooiiconui! \;ith ON'ration»< ros?onrch-' introductory, tito 
cnoc^ior •: v;':r r. «/!iro t'w sr .; v.c to ^tv.i!;.' .;r-">i^! .:'-:'/r- .:r Ion in t!u^ 0. \. area, :-:ot: 

int.ro;:;:tli^•/^ r: : iiv o: ^ro'*.:^i ! ic" ♦ r o.n intiwi-u C ion to 1 • :1 ♦•'tl^rn, 

got M ir.erov'',u*t i on to 1 i:^-,- v ^ .r.r I:);:, \:\d -ot o.n inrvo !::otion to 
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statistic::. T!u 40 Civc sub-oals arc the ha^is for .mastering the goal. In turn, the siihooals 
above are tlien treated separately, each as a subgoal to be encoded by itself. To satisfy the 
first suliioal ujove, the expert may encode the subgoal by giving the retrieval of Aronoisky' 
book as one possible satisfaction to this subgoal. Similarly, for the other subgoals. For 
exanplc, satisfying simulation (subgoal 3 ) may require knowledge about SDOJLA, as well as 
reading Balintfy's book. Hence, this subgoal is encoded again as a structure module, 
requiring (j.) introductory knowledge about simulation languages and (2') simulation 
applications. Then (1') is encoded with one particular solution, namely SIMUU, 
while (2') is also encoded with one possible solution. 

The distinction bet^Jeen structure and content modules is somewhat arbitrary, but 
convenient. Structure modules keep decomposing goals into subgoals until a level of 
subgoals is reached whereby there is some suitable material or physical resource that 
will satisfy each subgoal. Maintenance of this dichotomy be&,?een content and structure 
preserves the modularity of the system. We particularly do not want SBttJLA tied to 
simulation. Rather we want it tied to simulation languages. We do not want to go into 
the structure of the varied modules that use SIMULA to update them. 
Prerequisite Specification 

As explained above for a structure module, the encoder must specify a set of subgoals, 
the completion of which will satisfy the original goal. More precisely, this is a 
spetification of the sub-domains that comprise the subject matter in the area-part of 
Cht goal. It is analogous to a book's section headings, under a chapter that dealt 
with the area-part of the goal. 

It is only a diL'oction •.■»£ tli.' !:^aCorLal itself; it is not a ccroleto ela'i.iration of every 
piece of material that vould havo to bo specified such that t!'.e goal would be coniplotod. 
In this so:i.-io, subgoals ar.> vtct c':.' fano as proro<:v.isit«^s . (Tndood, to civo all nro- 
roquisit.s, ono would ii.iVc to spcviiy j reading wa;-a:>iiity, versatility in linglisU, cCc.'k 
TiiJt is, all ^onctM^"'* ro^'ii.siics n ajj o.v^jiiu*.! a-; >vMo-\\in^ ol tlu arva-p.irt 

gIv.M of l!:. ,v^.:I. Only cl.c- pro.vr su>Jivii5i.o..i t'.Mt b.-jv ^irc'cly o:i c'.k- 



area arc given; it is up to the expert to evaluate and properly pick such a decomposition. 
Naturally there v;ill be some variance among experts. This is one reason why more than 
one expert will encode the same areas. 

The place wlierc preliminary prerequisites do get included is in the content modules. 
When materials arc actually retrieved in order to satisfy the subgoals, the need arises 
for the preliminary prerequisites that will enable the student to handle these 
materials. In the portion of the content module form entitled ^'prerequisites" , such 
requiremonts are given. For example, assume that the student's goal is linear- 
programming/ introductory, C, 6-7, 3 v;eeks. 

Then the encoder must decide on the basic areas of an introduction to linear programming, 
given the other constraints in the goal. Let us assume he specifies the following subgoals: 

(1) objective^functions/introductory, C-B, 6-7, 1 week, 

(2) simplex-method/introductory, C-B, 6-7, 1 week 

(3) the-dual/introductory C-B, 6-7, 1 week 

The three subgoals above would be his choice for the decomposition of the goal. 
However, left unspecified are the prerequisites of sufficient calculus to understand 
(1) and enough linear algebra to comprehend (2) and (3), 

These prerequisites hc^i?ever are not really an integral 

part of linear programming per se. Thus they do not appear in (1) - (3^^ . [Perhaps some 
would say that some parts of calculus are to be included in the area of linear programming. 
Then they may have added a fourth subgoal of calculus with a time of fo'.ir months. This 
is pernissible, if the encoJo.r decides so. N*eedloss to say that he will have not encoded 
a stiMt'tcrrc r.u>.ii:!c chat vou.Ul ui! full t:!:o cvii;inr.l ^:ojl giv^rn^ bocauso of tire. Xc^ncthdoss 
it is a valid :r.oJulo. In .^livii a caso it sir.ply would rer^roscnt the best the encoder was 
able to Jo wit'', n tiVi* over:"! In this cj?o of four i-.onth??,] So let us as?:;'.r:e t!'.*:t the 
thrco sub-.;o»ils i;ivcn arc folt to lu» a propor sot, Wacn Clio subs;oal objoctive functions/ 
iatroduocory iy In in.^ or.oocv.i, c'.t-. cjKo.lcr r-ay oichov cliooso to iIoco;'.:u»so that lurtlur 
or ciioo.-»c j,*vciiy a cuuLcui ;..uwuic. A^su..;c thut he ciioo.sws a contcUL uodulc. Ihcu 
he indicates sot.io oducatioiiaL ro:»ourcc that adequately cuuxattucatos the concept. 
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Perhnpr. it is th'.* fir?t sod ion of the socoiu! chapter of the book Line "rot^r-irr-i! *./. 
The question posed in the encoder's fonu i$ what prerequisites are needed to handle this 
topic. It i.*^ hvre that the jncoJcr specifies the need for basic calculus. It is at 
this stage that he knows precisely what is needed, for he knows precisely what kinds of 
material th.c sti Jont V7ill be study ing. Actually one nay notice at this stage that it is 
not calculus at all that is needed. Rather it is an introduction to the simplicr concept 
of "functions", Uencc the encoder would be expected to specify this. 
Then again , perhaps the encoder nas picked a different book written especially 
for laymen. And this concept is excellently explicated without the notion of any 
higher mathctnatics; only high school algebra is needed. Then the prerequisites are so 
posed« Indeed the actual preliminary prerequisites can not be totally clear unless the 
actual material to be used is specified. 

As another example, imagine the goal of a two week introduction to relativity. The en- 
coder who puts tensor calculus as a prerequisite automatically, is quite mistaken; the ex- 
pert who chooses the content module may well pick a reformulation that requires nothing more 
than high school algebra (though an analytical mi.?,fn additon) . Hence such preliminary pre- 
requisites come into play when the actual materials are specified. Note that the pre- 
liminary prerequisites specified at such time may well be other goals, for example: 

calculus/introductory, C, 6-8, 2 months 
These goals in turn are re-entered into the system and are handled like any other goal. 

Note that the encoder may ( and frequently will ) 
alternate bet^ceon structure *nnd content modules. The original ^oal may be completely 
hnndlod by j bool: ho kn;>''s of. T!:v.i? thu initial orcocin^:; is one of contv-nC, not :^tr;:c- 
ture. liovevor, alter ypociTyinj suci: a book, he nay tvish to specify sovoral siibco;iN 
that are noce:isary in otvior thiit that book can be handled. These \\\ay include sever.ii 
general ^:o.tls i!t.^cribvJ in I'/^so in t;'rn /.re o-.^.tiv-ivd into t1:e .iysc-.:: 

and ^e \\r:t y.v bvo'.vn kl.v n b\ ev.c^i.H^ str-.u'Cure (or Cv^niiv.w •i\!i!!e>', yt i . . in>. 

Typie.-jJly t»-- bvp-.l.^r (or "hi;^»cr !vve!*M Vri^/!r!;^ <!e-r.:ti.^, trho r\\!- l.**-- all 'u- 

t* ti',»c Lilt %♦ .M L'.u« ii^M^^.-c K v\ 1 :> i.oXtVi-.w ly .';j^viiic ca.^Li-^ . ^ it vi.ll Lt.-iiu 

prii..arily eoutent modules. H*ViVi*r, this is not a requircr.ient. 
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Su:;r\ary 

In general, the encoder must decide 
whether the area-part of the goal, in context of the total goal, is best encoded by a 
structure or content nodule. (At the highest levels, he may be requested to er.ploy only 
structure modules). In the case o£ such structure modules, he will be asked to define 
the primary suhareas that coir.prise the don^ain of interest, given the time limitations, etc. 
These subtopics that define the curriculum for the student are expressed as subgoals to 
be accomplished. If the encoder then handles each subgoal in turn, which he may be called 
upon to do, he cycles again through this process. At such time as he feels that a 
particular educational resource precisely fits the needs of a sub • .*sub-goal under 
consideration, a content module is encoded. In encoding such a content module^ he must 
r.pecify the preliminary prerequisites so that the student can handle this material. This 
Is given as subgoals that must be fulfilled or met before there is a reasonable expectation 
that the material can be effectively studied and mastered. In general the encoder should 
strive to choose stiructure modules over content modules since this makes the area being 
developed less dependent upon specific educational resources. This point 
can be further appreciated by contrasting the encoding of the area '^management science" by 
listing 20 courses that must be taken^ versus listing 20 subgoals that must be fulfilled, 
each in turn requiring, say, 5 .sub-sub-goals, each one of which is fulfillable by either 
some chapiter, sone short series of lectures (perhaps on video tape), journal articles, 
or other educational media. Of course since content modules can have prerequisites that 
lead to other structure or co:;co:it ntodule*?, the curriculun ;^ill vary %^ith both conpononts. 
A hypotliot ic:*l curriciibr.n is itivcn in ficturo 2» as it ovoivod Cor the particular 
studonc'i; goal as i^hc/n. In the c-*..rricaU::-., che voriou» kinds of r.oJulc:*. aro oittlir.^c. 
Tho Ct r.'.i:"! il nv>.;os in the i'i .:r-.' r- »rcM«:i' p.^int At the curr iv*;:!;*:" .".■.t^'lu'S 

the crri f.i or c...n''» i 1 : w;. v>.' lli*- 5-trJ..:c, I'.u si* c-.ro all Liu- :^ u »it 

he may b*n;ln iituJy. UccauiJo oi space lir.iitntions , aiisOi-iatcJ r.i:!:5tory r.:.iccrial Is only 
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Further Specifications in Kncod jir.: M astc-ry and Time 

MastuTi?. W!>un encodins a module, the expert is asked to specify some evaluation 
or verification procedure that will enable the student Co confirn his mastery at 
the specified level in the goal. Such a determination of educational resources or acti- 
vities that adequately reflect the accomplishment of the appropriate mastery level 
involves a double consideration. The first and obvious one is what materials or tasks 
constitute a "test" or evaluation. The second and implicit consideration concerns what 
is to be tested. In the case of content modules, this second consideration is clear. 
The test is to concern thCj^atrerial referenced in the module itself. Left to be specified 
is the activity that constitutes a test or evaluation of the required mastery level for 
that module. In the case of structure modules, the material to be tested is partially 
obscured by the fact that the module is composed of sub-goals. However the mastery of 
the module is not of the subgoals; when they are encoded, the mastery of each one Is left 
to the encoder of each subgoal. Rather the mastery involves mastery of the "node" at 
Which the encoder is working. Thus the evaluation of mastery (at the specified level") 
requires evaluation of accomplishneht at the main node itself. 

For structure modules, it is swnetimes the -case that what the student is to get out 
or derive from the material contained in the subgoals is strictly greater than the sum 
of the parts. That is, there may be an integrating activity that unifies the separate 
subgoals which is an understanding over and beyond the content of each of the subgoals. 
It is often such an integration that constitutes the goal of the module Itself. Hence 
in those cases where the integration of the subgoals does not occur durina' the pursuit 
of the ^A'sor.ls, .'Vjch into-rati::- j.-tivi£y viU occur as the student atccr.;'ts to i!cr:on- 
stratc and evaluate his r.astory o: the ncdc itsolf. Such into-ratinj activity or catirial 
or r«.<.'.;rcc's -rt- ■• ♦ . ci f ; -.-d in : -.i:.c-.ry ^.czi.^-x of vh.^ i.-\:^cl-j :Vr-. 

The fii-..t portion oi c\w spc-ciiic.it ion o; tho ;-.:.i:.tory section :-.!.;y include a 
refcr<,:'.k-e to so-:,- --Jueauicn •.-■.";>•. :vv-i' thnt jtt. --'t:; to i-\t. ':\:tc .'.nd coor.i i-.ujto tho 

tlu« t-'.-iHltT f H v i: ic-ri SO!-.,- t,:-::: or ti-.t or :»rvv,..!uro w';, v.■^^• t'l.- -tud nt v.m- ovaiuat.-. 
hifi c.::ttt'ry of th. nodo vlth '.-h.fch ho i:; ctirro-.itlv involv.-.!. On tho ot'tor h.!!V,!,^ if in 
dolu); the du!.j,»...I.:, the .-.tuJuit. will acliicVi- tho ovorviow or unlfloatlon uri ho pvoj-.v.::. •• 
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through the subgoal list, such additional materials need not be assigned in the first part 
of the niastiV/ .^orclon for irh^- r;cuulc. The cholco ijnd decision i,s up to the encoder. 

Thus the r.istery portion^ is the section of the encoding format that contains not only 
descriptions of evaluative procedures, but also provides the opportunity (and in s<Kne cases 
the necessity) to assign explicit educational resources or activities that operate at the 
level of the ^oal itself (rather than any subgoal level). By way of analogy, recall that on 
final exams, professors will sometimes describe some short, comprehensive situation, involving 
an- integrative component among the sub- issues the student has studied. Then questions are 
asked about it. This is only partially analogo'js to the mastery section, 
of structure modules. Since we do not have the same time and space limitation, (nor, often 
the same motivation), we sljall replace the final exam's briefly described situation of inte- 
gration. In its place are references to something more akin to the last section of a book's 
chapter that attempts to pull all the sections of the chapter together. 
Associated with this assigned integrating activity are the evaluation 
procedures (or integrative final exam questions in our analogy) . In this respect, of 
course, both content and structure module*? are alike. As an example, we take the 

(partial) goal "programming AND statistics/intro'*. 

• • • 

Our structuring could take the forms as given in Figure 3(a). Here the intersection of 
these two areas is achieved in each of the subgoals. Mastery of the top goal requires 
only the union of the mastery of the subgoals. However, we might hive taken the 
approach found in Figure 3(b>. The student would have the building blocks from each of the two 

sub-ci\v:ain:;; ;K>\.\.Vi.r no activity yet vould intccrate t\w tvo seoaraco do:;:ains. Thus, in 
the r.3^«ten* t^orcicn of the tc> noio. (ori^:innl structure r.odulo^ , we would hnvo to sspocify the 
cduc»iLio:iaI i\ ^•.cul•v.•u5; thuc i;uU*4ii.u cjooin^; hvw prw*^ra;.:;;iny is applicu to statistics. 
Thort v!*' •a-.-..:-., ^--lo he r.i/.^.v* ♦ ^ .':.;v:vr:t: to cA c*^^ his c*<: '/ot* •* jo ut i^orj r;./: ivry Kv-**U, 



9 



3 (a) 



pr'ogranjnins 
significance- tests 
V introductory : 



prog-rcn'ming and statistic 
introductory 
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3 C 



programTiing AND anova 
introductory 



programming AND factor-analysis 
introductory 



5 



Computer Programming 
in Statistics 



(chapter 1: 
significance tests) 



Computer Programming 
in Statistics 



(chaptier 2: ANOVA) 



Computer Programming 
in Statistics 



(chapter 3: 
factor analysis) 



/programming and statistics 
introductory 



3 (b) 



c 



programming 
Introductory 




See last section of each 
of chapters 1,2 and 3 
of Computer Programming 
in Statistics 



Programming 
Made 
Simple 
Chapters 1,2^3 



Statistics 
Made 
Simple 
Chapters 1-4 



c 



Symbols 
^\ • structure modules 



* content modules 



• assigned integration 
(mast€ry)material 



Figure 3 



Similar curricula with and without integrative material 
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Tim o Rocall that the student is assumed to be working full time exclusively on 
the particular goal being encoded (or each subgoal in turn) . The estimate of time needed 
to complete a module includes the time needed to complete any Integrative assignments 
given in the mastery section. Belo^^^ are some approximations that relate university 
courses and terms to such lull ti:.ie e££ott« 

a) A3 credit cotirse per sorester is npproxinately equivalent to 4 weeks at full- 
tine (or one month> 

b) A 2 semester course equals 2 months. 

c) A A credit course equals 5 1/2 weeks or 1 1/3 months. 

d) A mini course at G.S.|UA. is equivalent to 2 1/2 weeks. 

3) Six weeks of a normal term in a 3 credit course is equivalent to 1 1/2 weeks. 



Some Other Suirjnaries 

i 

Level: introductory - 1 
intermediate 2 
advanced - 3 



Mastery: 



Level 1 (Pass) 

Level 2 (C) 

Level 3 (B) 

Level 4 (A) 



Motive: I - lowest 



9 - hishost 

CfOAL: Aroa-part/Lovcl, Mastory-Rvinso, Motive-range , Time-Range^ 

/ 

••£dontlc.tIl>-": ■ / ■ /' 

> . 

(Yk'u r...iy wifh to t. ..r t!t;:. iV..,r c\i to k.'op .i-; rvi\ frnc-^ 



Appendix B 



coaTi:::? r:or;i*Ln i:::codt!:g i:;FTrrcTior:s 

0. Introduction 

Xha purpose of these instructions is to aid you. in filling out the encoding 
form. You will find both new information as well as sun;xnaries of some information 
that was in the Encoding Guide^ which we assume that you have already read. In 
•particular, we assume that you are familiar with a GOAL statement and understand the 
purpose of the prerequisite lists in content modules as well as the distinction between 
content structure modules* Checking each number and subsection of the encoding form 
with these instructions should facilitate your encoding task* In the upper left 
hand (portion of each form, you will find a place to number (in any fashion) each 
module as well as give your name; in the right hand corner^ you may refer to an 
earlier module, which will imply that this module is identical to the one so referenced, 
except for those sections that you fill in (which will override the sections in the 
referenced module) • This is a "ditto" box, for your convenience, when you find ypiu 
are encoding the same thing, or same section, over and over. The rest of the guide is 
self*explanatory. 



B-2 



1. y.oJu le ^NA/'.K; 

a. An identifying name 

Choose a very sin:ple nama in natural language format to describe 
this module. ( .g. , Wilde* s introductory OR book, "The Brain" 
film, Samuelson's economics book.) 

b. Abstract 

In a few sentences, give a brief description of the module. 

c. Condense the abstract above ir*-^ one single, terse sentence. 

d. Area-part of GOAL 

Translate « the above sentence into an area-part of a goal (in 
GOAL language). Review GOAL in the "Description of Your Task** 
document if necessary. Do not include a specification on 
Level; that will be treated separately. 
' e. Semantics 

For each word (w) used in (d) above, give three lists. The- 
first list (Aj^) contains all words which imply w, or from which 
V is an immediate subpart or subset. In a knowledge domain. 
It is the category just above the category that contains w. 
The decision of Aj is based on the use of w in (c) , and hence* 
in turn on the concciit and context of the module being enccnicU. 

Then give list A,. This containi« all words equivalent to w»r 

sy^vW ^'\: . Iz i : \'Wi^: a:: t'^v'^ri.t; wtth 

w, tlic sa*.c c*uc u^ry a.^ 

Finally t^v^Ji- ic:* l.u»>o v*.*ru.< i cau .*ri* -^ 

that Av*^ *; \*^\: iry** tro v V**' • t r v. 
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Tend to lir.it tho numher of entrios of each to ap-^rozzimatcly three or the 
total (of all nter.bcrs of all the A^'.n» to nino. This rule is not fixed. 

If neccspnry, re-arrangc snncc (s) to fit larj^o lists, by boxing in trore lines 
vherc ncuiicd. If more spaco. is needed for other words, use the back of the first 
page* There is a box to check on the first page if you do this. The same Is true 
for the subgoal list on the second page. 

For example, if operations- research had appeared in section (d) , then section (e> 
might have: 



> Appltod-nath 
( Management « sc icnc e 




Operations- Research ^ A2 = Quantitative-methods 



r Dynamic-prograrnnine 
Linear*-r»rograr:niing 



Non 1 i nea pvop, ramr. in«j 

t - J 



Simula-: .on 
Prchabllitv 



1 

Statistics 
Quouing-theon^ 
Control-^tbeorv 

Note the use of the hyp!;iin. This Is to er.Miasize that such word pairs as 
'•operutior-i- rvi^tarci:*' Jinu "co'v:;Kx vari.::>i;.':s" cro vcaily to be con5ii*u.rcd sin^U- 
terns. T?:o conposito vord (e*j:., co"-;^lo:-:-v;:r Lnblo.^^ constitutes a sinyrlo torr 
For vory -rr.^ ral \.\^vd^ v-;':Mrc/.::-.it-i^cionco, p!iilo:-.o:>h\ ^ , thorc nay he no 

logical up-. L- noi^i.i'v-r.:.. J. In .^uch casi .^^, ciic vorJ ils^-lt is the only r.onbor 
of tho Aj .-wC. rur c\..\^!i-, t!io Aj M-t *V^:nj u vu nt-^clvnci-** contains only t\w 
ter:;: **t.:-ina.,vr.xnt i»cicuvc*; tho Aj set lor .•.4ilo:iOj)ay ' contains anly tho. word 
"piiilojioph\*\ 



2. Level 

Choose whether this module is to bo considered an Introductory, inter- 
mediate, cr advanced one. Your choice will depend on your purpose in 
encoding it. Other criteria can affect your choice. For example, the 
same module n;ay be encoded introductory or intermediate depending 
on the length of tine snent on the module. Abbreviations for levels 
are: introductory = 1, intermediate = 2, advanced « 3. if you wish, 
you may specify a range. 

3. Time ' 

The remarks in "level" above hold here too analogously. In giving the 
time, specify the range. The first box allows for a number and a measure 
ment (i.e., hours, days, etc.) as does the second box which gives the 
upper range. Try to use the same measurements in giving the lower and 
upper ranges (e.g., "1 day - 3 days" rather than "8 hours - 3 days"). 
This is not a fixed rule.. 

4. Media 

Check off the m^ia in which this module is encoded. If "other" applies, 
place a message in quotes that tells how the material is encoded. 

5. Motive 

Give two numbers which describes the range of motivation, ranked from 
one to nine, within which this material is suitable for use. One Is 
lowest and nine Is highest. Thus, If the material is suited for only 
the tncs t scrivM-s stud^MiG^; one mlc:ht choose the range 8-9. A very simple 
"popular i::Atioii" may be n:orc suited to 2.wuJ:cr.:5 vitli little soriousnc-:;s 
of {Mirpo-^o (L.o., 1-3). ::'atcrial i:uy bo appropriate lor o>;tiv.r:oly hiVi'd 
(o.g;, or n;.-ro^: r.-:^^:: u.;:., '^9--**, ♦'l-r*, **3'3-*>. 



0 
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6. Access 

Chock Che box that Indicates how one may get the module for use* If the 
box •'other'^ is checked, ?^\vq. a message in quotes that explains how one 
can access the material. 

7. Citation 

a. Formal description 

A .formal (bibliographic-like) citation of the module is required. 

b. Evaluation 

A number reflecting a measure of the quality of this module (from 
one as the lowest, rated "quite poor" to ten the highest, rated 
••excellent"). Naturally, the goal that this module is to serve 
will directly affect the choice here. 

8. Prerequisites 

a. Kducatior: Icw.l ncci-.d 

Choo5o the nur..ecr corrcs-.^oiullnf; to the hi^.hest level needed 
bcfovc crtc can "hr.:iJlc" tiilis r.odulu. This is often r.n estimate 
of ••sopisisriccitii^n" necdc^!* 

b. Prerequisites needed 

Give a complete GOAL statement that describes the prerequisites. 
Tfiis soal described is tlie one that the s;tudent neec*i? to fulfill 
to be a'-'le to aecopt .'u^ ittoJule. 1'hcre ::uy be sc/i^r.il .;»>a!5 
nJco:5s:ry: 1 I.-;t eich iJoparaCelv, If or.e -:oal opvi ni.^lv .mJ 

Ihis is r.ot a requirement. 
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Note that you aio to estimate the suggested time, 
motive, and mastery ranges that suitably describe the subgoals 
as you intend then. The total time of all the subgoals should 
not exceed the tot..l ti^ie alloted for this module in part 3 
above* By specifying ranges rather than exact times, you will 
.eliminate the need for constant checking of the total time or 
petty number- juggling. Also there may be some correlation between 
the motive of the module itself and the kinds of motives to be 
expe<jted in the pursuit of the subgoals. Of course, there may be 
no relations, exceptions, etc. There is no fixed rule involved. 
Yoi* may use a range for each parameter, and hence, can include and 
exclude whatever you wish. 

c. Aptitudes 

If any of the standard tests listed are pertinent, give the 
minimum scores needed or expected in the respective boxes* If 
another test is required, give its name and minimum score in 
the extra space, as provided (in quotes). 

d. Attitudes 

If any particular attitudes are required* for the module, either 
tests and minimum scores may be listed or quoted material entered. 

e. Other 

Give in quotes viny prerequisite tiiac must be mat chat coulo. not 
be cnCiVd /;^ovo. I'l;!.'; r..o:5:5a;:e rocurnod :o c.!l» user as 

an ini-jiiir;. ; a ^o^^ 'no ros/..M:so U' v\ will Jol. wiu^: 
he rvH^t.-? the o.Mviition. A •'no'' tvill cuu^e the rwlvAc to bo 
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Mastery 

a. There arc four rastcry levels, anolgous to a student detnon- 

strating co:.*.petoncc in sor.e course with either a grade of "pass^' 
(level 1), a grade of ''C (level 2), a grade of "IV (level 3), 
or a grade of ''A" (level ,4). Specify the level (or range of 
levels if this applies), for which this module is suited. Make 
an effort to achieve agreenent between the goal of this module 
and this level indicated (or range specified), 
b. Specify evaluation or testing procedures that would 

adequately reflect a competence at the mastery level (s) specified. 
This portion too will be quoted. Be clear and coTiplete. 



Mociulo Number 
(plus your 
name 



Module tncodmg torra 
Content Modules 



1. Module NAME 

a. Identifying Name 
Abstract : 
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This module is the same as 
module 

In sections 



c. In natural language:^ 
(a sentence) 



d« Given in terms of the 
AREA*- part of the GOAL 
language: 

e« Semantics: 



A^ • 
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Check hero if <>5her words 
arc m back ! 1 



i 
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2. LEVEL ! 



(a number or range where introductory « 1, intermediate » 2» 

advanced « 3) 



3. TIME (for completion): 
minimum 



maximum 



4. MEDIA (in which module is encoded): 

i books 1 course I film ( audio \ video-tar>e 1 computer I loumal I semi nar ) consultation 1 
other : 



5. MOTIVE: 

material raltable for students with motives ranging from to (range [1^9] 



6. ACCESS: 



mt library | Science Library ^ bookstore | audto/vlaual roomj 



other: 



7. CITATIOW; 

a« Formal description of material:. 



b* Evaluation (Score from 1*10, where I » quite poor, . * • 10 • excellent): 



8. PREREQUISITES; 
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A. Educational level 
needed for module 

b. PREREQUISITE LIST; 



high school 
0 



colleee 
1 12 1314 



graduate 



Area/Level 



Mastery Motive 
(range 1-4) (range 1-9) 



Time 
range 



2. 



5. 



6. 



7. 



cheek here if other prerequisites 
are on the back 



□ 
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c. Aptitudes: 



GRE 



SAT 



IQ 



ATGSB 



OTHER: 



d. Attitudes: 



c. Other (any material of important nature allowed, to be answered "yes" or 
"no" by the student) j 




"9r~M^fERY 



a. Mastery Level | j ( a number or range between 1 and 4) 

b. Mastery Tests 



Appendix C 



0« Jntrcductlon 

The purpose of those instructions Is to aid you In filling out the encoding 
form. You will find both new information as well as summaries of some information 
that was in the Encoding Guide, which we assume that you have already read. In 
particular » we assume that you are familiar with a GOAL statement, understand the 
purpose of the subgoal lists in structure modules as well as the distinction between 
structure; and content modules, and are familiar with the intent of the mastery sec* 

if*"""*** * 

tlon (giving integrative information when necessary as well as testing procedures). 
Checking each number and subsection of the encoding form with these instructions 
should facilitate your encoding task. In the upper left hand portion of each form, 
you will find a place to number (in any fashion) each module as well as give your 
name; In the right hand comer, you may refer to an earlier module, which will 
imply that this module Is Identical to the one so referenced, except for those 
sections that you fill in (which will override the sections In the referenced module) . 
This is a ••ditto*' box, for your .convenience, when you find you are encoding the same 
thing, or same section, over and over. l'*e rest of the guide Is self-explanatory. 
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!• Module Nnnie 

a. Choose any convenient simple name, in natural lannunge, for 
referring to tuis module. (This may be the same as part (b) 
or (c)). 

b. In natural language, give the purpose or goal of the module; 
this may often be simply the goal statement of the module, 
part (c), or even part (a) again. 

c. Give the .goal thac this module is to satisfy in terms of the 
area-part of the Ga\L language. Again, this could be the 
same es part (a) of (b). 

Semantics 

For each word., or hyphenated terms (w) used in (c) above, 
give three lists. The first list (A^) contains all words which 
imply w, or from which w is an immediate subpart or subset. In 
a knowledge domain, it is the category just above the category 
that contains w. The decision of A^ is based on the. encoder's 
intended use of w, and hence, is based on Che content and context 
of the module being encoded « 

Then give list A^. This contains all words equivalent to or 
synononious with w. It is those words on an equal footing with 
w, the same category as w. 

Finally encode A^ which are Chose words (i.e., caCescrics) 

tluit aro s;!>.-H*r*? cf or follow fro/: or -^re derivable iror.: w. 
It is t!*e cnTo;^jn/(s^ just bclov: ct*v* ri*pro<?on::od by w. 

Tt-nd CO lir-.i: tlie nj-bor oi* ontriws of cac!i to aivnvMi/aicly 
thrci- or cho cotal (of all r/.cnibcrs of all Chu A^'s) to nine. 
This rulo is not :t :td. 
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If III ci'?.;::«ry , Vr-nrrauri- (y.) to fit lat:j*.o Ufit;;, by hoxlnf. in uoro lincfi c 

wlu*re neodfd. If more spj.co. is needed for otlicr v;ords, ur.c the back of the first 
pri^i-. Tl-.v i;. .i !>0': to <./ ol; on i!k fifJC p:':*.c if you do Ihir.. The 'm\::\c is tnu 
for the subgocil lint on the r^ccond pn\v. 

For cxr;:-^>le, if oporations-rc;iearcli hc-d appeared in section (c ) , then section 
might have; 

I Applied ':*:atlt 
Mann rror^on t pc icncc 

w « pporation?-?.os v-^rch ^ - Quan t i tat ivc -no t hod s 

I D>mar;ic-nror.rar.T» i ng 
Llnear-prorrar.Tr>ing 
Nonlinear-nropraTnmin^; 
' Simulation 
Probability 
Statistics 
Queuing-th.cor^/ 
C'-.*n t ro I— tf^'^^orv 

Note the xxsc of the hyphen. This is to e:?.phasi2c that such word pairs as 
"operations rw'scatch" and *'co:nple>: variables" are really to be conbi'^^ rod single 
tcr»r5* The cc;'.:;>csite word (e.g., CG:i;lc!:>varir*:>U',<^ ccnttitutes a sitvr!v: ternV. 
For vcvy ,7oni'ral \.*ord:: v\.\v.:*;\',': nt-^cioncc-, p!ulci?o;.5:y'^ , tlicvo r.*:v bo no 






C-4 

2. Choose whether this module is to be considered an introductory, inter- 
p' r!lntf, or advanced one. Yoitr choice wijl depend on your purpose in 
encoding it. Other criteria can affect your choice. For exa^iple, the 
sanie module may bo encoded introductory or intermediate depending on 
the length of time spent on the module. Abbreviations for levels are: 
introductory = 1, intermediate = 2, advanced = 3. If you wish, you may 
specify a range. 
• 3. Motive 

Give ti^o numbers which describes the range of motivation, ranked from 
one to nine, within which this goal is suitable for use. One. is lowest 
and nine Is highest. Thus, if the goal Is suited for only the most 
serious student, one night choose the range 8*9. A very simple approach 
may be more suited to students .with little seriousness of purpose. 
(I.e., 1-3). The goal may be appropriate for extremely broad (e.g., "1-9") 
or narro«r ranges (e.g., "9-9", "1-1", "5-5"). 
4. Subgoal List 

Give the various subgoals whose accomplishment will satisfy the goal 
of this module. Note that you are to estimate the suggested time, 
motive, and mastery ranges that suitably describe the subgoals as you 
intend them. The total time of all the subgoals should not exceed the 
total time alloted for this module In part 3 above^ By specifying 
ranges rather than enact tines you will eliminate the need for constant 
checking; of the coLal tlv.^ or potty ni:r.>cr- jj;:^;! ing. Also t::oi*o r,;v.' be 
^orr.c correlation bccvoon the r.otivo of the n«.ovIulo itself itnJ the kinds 
oi nor Lvc5 to \w. «. xt^wctod in ciiO* pufr*.r.5it oi the Miiji;o::i». ^-oi-.r^ie. 



You may use a rnn for each paratictor, and hcnrc, can include and ^,^5 
exclude vfhatovcr you wish. If some subgoals would best be started after the 
completion of other subr,^nls, put thoso donondont subgoals after the more 
"preliminary" ones. When estimting time for completion of some module, . 
assume the student is working full- tine exclusively on that module. It 
is his only cask to be pursued throughout the whole work-day. 
The estimate of the time to complete a module includes the time needed 
to do any integrative assignments given in the mastery section. To convert 
"semester courses" into full-time efforts (p'.us other equivalences) use 

the following guide: 

a. One 3-hour course = 4 weeks or 1 month of full-time effort 

One 4-hour course = 5 1/2 weeks or 1 1/3 months 
c. Six weeks o£ a course = I 1/2 weeks 
d« A mini-course ^2 1/2 weeks 
$• Prerequisites 

a. Education level needed 

Choose the aumber corresponding to the highest level needed 
before one can **handle" this module. This is often an estimate 
of "sophistication" needed. 

b. Aptitudes 

If any of the standard tests listed are pertinent, give the 
minimum scores needed or expected in the respective boxes. If 
another test is required, give its name and minimum score in 
the excra 5:v.co, cs i-^rovidcd (in quotcf ) . Vou - ry use values 

or percent ilcs . 

c. At::lLi:J s 

Ii any partiv-ular 4i:lu.;Js.5 ar*. rc.iuivcJ lor tito -.nt.iU*,.! I'O'.'t.^tu^ 



d. Other 

Give (in qttotes> any prerequisite that must be met that could not 
be encoded above. This niossage will bo returned to the user as 
an inquiry; a yes/no response from him will determine whether 
he meets the condition. A "no" will cause the nodule to be 
rejected as unsuitable, 
tery ^ 

There arc four iTtastery levels, analogous to a student demonstrating 
ca-npetence in some course with either a grade of "pass" (level 1), 
a grade of "C" (level 2), a grade of "B" (level 3^, or a grad.e of "A" 
(level 4). Specify the level (or range of levels if this applies), 
for which this module is suited. Make an effort to achieve agreement 
between the goal of this module and the mastery level you specify. 

Give suitable educationa'. resources or activities that may be 
necessary to integrate (or summarize or consolidated the collection of 
subgoals specified in part 4. Thus this material is at the level of 
the goal itself, not any sub goal. If you do.not feel this is needed 
or is already achieved in the accooiplishment of the subgcals, you may 
omit this section. If an entry is made in this section, it will be 
outputtcd to the student exactly as it appears (It is merely quoted^ , 
Bo clcnr, ccr.plfto, and give conciHv^ rjfcr.^nccs \%*horc neccsi^nry. This 
is the only ir.torution c!ie studont will have. 

Bajjed on part (b> above as well as the subpoals thorwolvos, sncciiy 
cvaluuiLon cr ti s»cin^ procLJurc^ that \:jaU\ adequately vciluxt a cct::- 
peter.ce ^;t: tlic r.AiXtcf: levd (s^ j>,n'Ciiu'J. This puillon Tc^o 
rtt-'^t.'v!. "0 clcrx r.::.! c/">lcti^ Tb.o toftr .cvnUu:tc tvi'tu^rv of tlu* ^-il 
ratlu r tiuiu jut't Cno i^;*; .,ouIt> , tiU'Ujh th^re arc ci;.;wb wa^a ttxij u;a> 
be equivalent. 



Module Encoding Form: 
Structure Modules 



t.:.^!ule Number T 



^p..us your 
name 



This nvodule is the 

same as module 

in sections 



1. Module NAMg 

tt. Identifying Kan\e: 



1 



b. In natural language, give Che goal of the module: 
(a sentence) 



c. Clve goal in terms of the 
AREA-part of the GOAL 
language: ^^^^ 



d. Semantics: 

— t *i 



*2 • 




'heck here if other words 
on hfick 



c-8 



I (a number or range where introductory - 1, intermediate - 2, 

advanced =3) I 



MOTIVE: Completion of module appropriate for students with motives ranging from 
^''^ I 1 ^° I 1 (range [1,9] 

SUBGOAL LIST: 

Area/Level Mastery Motive Time 

(range) (range) (range) 

1. 



3. 



ChecW here if othtr sub-goalt art gii b«ck| [ 



GENER/M. PREllEQUrslTES: 



a. Cencral educational level needed: j high schoo l 

b. Aptitudes: 1 CRE 



c. Attitt:des: 



colle?.e 

l!2bl 4__ 



graduate 
516171 8 !iJ 




d. Other (any material of important nature allo^d, to be answered "y" or W 
by the student): 



Mastery 



a. Mastery Level 



(a ntuaber or range between 1 and 4) 



b. Maftery Assignment 



c« Mastery Tests 



